This code implements the GPstruct learning algorithm:
The code has been tested with Python 2.7.8 and 3.4.1, Numpy 1.8.1, Numba 0.13.3, on Ubuntu 12.04.
There are basically two groups of code files:
Chain data: we used 4 tasks from the CRF++ dataset. The .x files have got sparse feature vectors for each position in the chain (ie word). The .y vectors have got the reference labels for each word. So my program is not concerned with feature extraction at all.
learn_predic_gpstruct ca line 218, computation of f_star_mean: Note that because of eq 4 (arXiv preprint), the kernel matrices need to contain only data for k_x, not for k_u (which contains k_x blocks on the diagonal). Therefore when eg multiplying with an f, you need to do as if you were repeating the kernel matrix on the diagonal, and multiply by blocks of f every time (iterating over labels y_t).
Sampling from f* (as opposed to taking the mode/ mean f*_MAP as now) is not implemented yet, it doesn't seem to bring a performance improvement, cf arXiv paper.
Not implemented either at this point (working on it): hyperparameter sampling.
The ICML paper requires splitting up training and prediction in order to do ensemble learning. That's not implemented yet. It requires being able to compute pseudo-likelihoods on grids where only some pixels are seen (ie there are occluded pixels); that's implemented in the PL code (cf visible_pixels argument).
hashable_compute_kStarTKInv_unary: will cache the computation of this kernel matrix, in case it can be re-used for a subsequent run on the same data, if it is still in memory.
pygpstruct_location = '/home/sb358/pygpstruct_demo'
! rm -rf /home/sb358/pygpstruct_demo
! cd /home/sb358/pygpstruct-master
! mkdir /home/sb358/pygpstruct_demo
! git archive master | tar -x -C /home/sb358/pygpstruct_demo
%load_ext autoreload
%autoreload 2
import sys
sys.path.append(pygpstruct_location + '/src/') # replace by your path to .py files
import prepare_from_data_chain
n_data=10
!rm -rf /tmp/pygpstruct_demo
prepare_from_data_chain.learn_predict_gpstruct_wrapper(data_indices_train = np.arange(n_data), # training data= first 10 data files
data_indices_test = np.arange(n_data, n_data*2), # test data= next 10 files
result_prefix='/tmp/pygpstruct_demo/', # where to store the result files
data_folder = pygpstruct_location + '/data/japanesene/',
n_samples=501, # how many MCMC iterations
task='japanesene',
prediction_thinning=100
)
2014-10-23 12:06:54,198Z - INFO - learn_predict_gpstruct started with arguments: {'kernel': <function kernel_linear_unary at 0x7f3c92ae3ed8>, 'console_log': True, 'random_seed': 0, 'prediction_verbosity': None, 'n_samples': 501, 'hp_sampling_thinning': 1, 'hp_sampling_mode': None, 'hp_debug': False, 'prior': 1, 'lhp_update': {}, 'prepare_from_data': <function <lambda> at 0x7f3c8d554c80>, 'stop_check': None, 'n_f_star': 0, 'prediction_thinning': 100, 'result_prefix': '/tmp/pygpstruct_demo/'} 2014-10-23 12:06:54,202Z - INFO - prepare_from_data_chain started with arguments: {'data_folder': '/home/sb358/pygpstruct_demo/data/japanesene/', 'data_indices_train': array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]), 'n_labels': 17, 'native_implementation': False, 'data_indices_test': array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]), 'n_features_x': 102799, 'logger': <logging.Logger object at 0x7f3c8e3541d0>} 2014-10-23 12:06:54,338Z - DEBUG - prepare kernel matrices 2014-10-23 12:06:54,351Z - DEBUG - start MCMC chain 2014-10-23 12:06:56,588Z - INFO - ESS it 0 -- LL train | last f = -675.36 -- test set error | last f = 0.79149 -- LL test | last f = -602.05 -- test set error (marginalized over f's)= 0.79149 -- average per-atom negative log posterior marginals = 2.0468 2014-10-23 12:07:04,106Z - INFO - ESS it 100 -- LL train | last f = -374.95 -- test set error | last f = 0.24681 -- LL test | last f = -201.49 -- test set error (marginalized over f's)= 0.24255 -- average per-atom negative log posterior marginals = 0.68085 2014-10-23 12:07:11,032Z - INFO - ESS it 200 -- LL train | last f = -277.82 -- test set error | last f = 0.18298 -- LL test | last f = -156.46 -- test set error (marginalized over f's)= 0.13191 -- average per-atom negative log posterior marginals = 0.37021 2014-10-23 12:07:18,258Z - INFO - ESS it 300 -- LL train | last f = -209.34 -- test set error | last f = 0.11064 -- LL test | last f = -120.9 -- test set error (marginalized over f's)= 0.089362 -- average per-atom negative log posterior marginals = 0.29787 2014-10-23 12:07:25,529Z - INFO - ESS it 400 -- LL train | last f = -181.66 -- test set error | last f = 0.13617 -- LL test | last f = -124.55 -- test set error (marginalized over f's)= 0.097872 -- average per-atom negative log posterior marginals = 0.27234 2014-10-23 12:07:32,658Z - INFO - ESS it 500 -- LL train | last f = -138.44 -- test set error | last f = 0.1234 -- LL test | last f = -123.18 -- test set error (marginalized over f's)= 0.089362 -- average per-atom negative log posterior marginals = 0.25957
I originally implemented the forwards-backwards algorithm used to compute the likelihood in the MCMC algorithm used for training in Numba. Because this is the speed bottleneck for training, I have reimplemented it in C as a Python module, which needs to be compiled separately:
! cd /home/sb358/pygpstruct-master/src/chain_forwards_backwards_native/; python setup.py install
running install running build running build_ext running install_lib running install_egg_info Removing /home/sb358/anaconda/lib/python2.7/site-packages/UNKNOWN-0.0.0-py2.7.egg-info Writing /home/sb358/anaconda/lib/python2.7/site-packages/UNKNOWN-0.0.0-py2.7.egg-info
Then the native implementation can be used instead of the default Numba implementation by passing the native_implementation=True argument; that makes the run much faster:
n_data=10
!rm -rf /tmp/pygpstruct_demo
prepare_from_data_chain.learn_predict_gpstruct_wrapper(data_indices_train = np.arange(n_data), # training data= first 10 data files
data_indices_test = np.arange(n_data, n_data*2), # test data= next 10 files
result_prefix='/tmp/pygpstruct_demo/', # where to store the result files
data_folder = pygpstruct_location + '/data/japanesene/',
n_samples=501, # how many MCMC iterations
task='japanesene',
native_implementation = True,
prediction_thinning=100
)
2014-10-23 12:07:34,849Z - INFO - learn_predict_gpstruct started with arguments: {'kernel': <function kernel_linear_unary at 0x7f3c92ae3ed8>, 'console_log': True, 'random_seed': 0, 'prediction_verbosity': None, 'n_samples': 501, 'hp_sampling_thinning': 1, 'hp_sampling_mode': None, 'hp_debug': False, 'prior': 1, 'lhp_update': {}, 'prepare_from_data': <function <lambda> at 0x7f3c8d554ed8>, 'stop_check': None, 'n_f_star': 0, 'prediction_thinning': 100, 'result_prefix': '/tmp/pygpstruct_demo/'} 2014-10-23 12:07:34,853Z - INFO - prepare_from_data_chain started with arguments: {'data_folder': '/home/sb358/pygpstruct_demo/data/japanesene/', 'data_indices_train': array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]), 'n_labels': 17, 'native_implementation': True, 'data_indices_test': array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]), 'n_features_x': 102799, 'logger': <logging.Logger object at 0x7f3c8e3541d0>} 2014-10-23 12:07:34,969Z - DEBUG - prepare kernel matrices 2014-10-23 12:07:34,982Z - DEBUG - start MCMC chain 2014-10-23 12:07:35,091Z - INFO - ESS it 0 -- LL train | last f = -675.36 -- test set error | last f = 0.79149 -- LL test | last f = -602.05 -- test set error (marginalized over f's)= 0.79149 -- average per-atom negative log posterior marginals = 2.0468 2014-10-23 12:07:36,132Z - INFO - ESS it 100 -- LL train | last f = -374.95 -- test set error | last f = 0.24681 -- LL test | last f = -201.49 -- test set error (marginalized over f's)= 0.24255 -- average per-atom negative log posterior marginals = 0.68085 2014-10-23 12:07:37,105Z - INFO - ESS it 200 -- LL train | last f = -277.82 -- test set error | last f = 0.18298 -- LL test | last f = -156.46 -- test set error (marginalized over f's)= 0.13191 -- average per-atom negative log posterior marginals = 0.37021 2014-10-23 12:07:38,099Z - INFO - ESS it 300 -- LL train | last f = -209.34 -- test set error | last f = 0.11064 -- LL test | last f = -120.9 -- test set error (marginalized over f's)= 0.089362 -- average per-atom negative log posterior marginals = 0.29787 2014-10-23 12:07:39,094Z - INFO - ESS it 400 -- LL train | last f = -181.66 -- test set error | last f = 0.13617 -- LL test | last f = -124.55 -- test set error (marginalized over f's)= 0.097872 -- average per-atom negative log posterior marginals = 0.27234 2014-10-23 12:07:40,221Z - INFO - ESS it 500 -- LL train | last f = -138.44 -- test set error | last f = 0.1234 -- LL test | last f = -123.18 -- test set error (marginalized over f's)= 0.089362 -- average per-atom negative log posterior marginals = 0.25957
import util
util.make_figure(np.arange(5), # display all 5 collected types of data
[('Japanese Named Entity Recognition task', '/tmp/pygpstruct_demo' + '/results.bin')], # can display several results on same plot by appending tuples to this list
bottom=None, top=None, save_pdf=False, max_display_length=2500)
file_pattern: /tmp/pygpstruct_demo/results.bin matching files: ['/tmp/pygpstruct_demo/results.bin'] file_pattern: /tmp/pygpstruct_demo/results.bin matching files: ['/tmp/pygpstruct_demo/results.bin'] file_pattern: /tmp/pygpstruct_demo/results.bin matching files: ['/tmp/pygpstruct_demo/results.bin'] file_pattern: /tmp/pygpstruct_demo/results.bin matching files: ['/tmp/pygpstruct_demo/results.bin'] file_pattern: /tmp/pygpstruct_demo/results.bin matching files: ['/tmp/pygpstruct_demo/results.bin']
The MCMC chain's state is saved at every MCMC step; that includes the state of the pseudo-random number generator, the last f, the current log likelihood, the marginals so far etc.
The MCMC chain can be interrupted between steps. However, there is no mechanism to prevent it from being stopped while it writes the state files to disk, which would cause the state to be lost. Interrupt the MCMC chain can be done thus:
When restarting with the same results directory, the program will check for the existence of saved state files. If it finds them, it will restart from the last saved state and continue the chain. It is the user's responsibility to ensure that the saved state corresponds exactly to the configuration of the experiment that is restarted. For instance, the training and test data should match, and the various other parameters passed to learn_predict_gpstruct should match as well.
For example, here we run the chain for 10 samples:
n_data=10
!rm -rf /tmp/pygpstruct_demo
prepare_from_data_chain.learn_predict_gpstruct_wrapper(data_indices_train = np.arange(n_data), # training data= first 10 data files
data_indices_test = np.arange(n_data, n_data*2), # test data= next 10 files
result_prefix='/tmp/pygpstruct_demo/', # where to store the result files
data_folder = pygpstruct_location + '/data/japanesene/',
n_samples=10, # how many MCMC iterations
task='japanesene',
prediction_thinning=1
)
2014-10-23 12:07:41,150Z - INFO - learn_predict_gpstruct started with arguments: {'kernel': <function kernel_linear_unary at 0x7f3c92ae3ed8>, 'console_log': True, 'random_seed': 0, 'prediction_verbosity': None, 'n_samples': 10, 'hp_sampling_thinning': 1, 'hp_sampling_mode': None, 'hp_debug': False, 'prior': 1, 'lhp_update': {}, 'prepare_from_data': <function <lambda> at 0x7f3c8c794938>, 'stop_check': None, 'n_f_star': 0, 'prediction_thinning': 1, 'result_prefix': '/tmp/pygpstruct_demo/'} 2014-10-23 12:07:41,154Z - INFO - prepare_from_data_chain started with arguments: {'data_folder': '/home/sb358/pygpstruct_demo/data/japanesene/', 'data_indices_train': array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]), 'n_labels': 17, 'native_implementation': False, 'data_indices_test': array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]), 'n_features_x': 102799, 'logger': <logging.Logger object at 0x7f3c8e3541d0>} 2014-10-23 12:07:41,258Z - DEBUG - prepare kernel matrices 2014-10-23 12:07:41,270Z - DEBUG - start MCMC chain 2014-10-23 12:07:41,351Z - INFO - ESS it 0 -- LL train | last f = -675.36 -- test set error | last f = 0.79149 -- LL test | last f = -602.05 -- test set error (marginalized over f's)= 0.79149 -- average per-atom negative log posterior marginals = 2.0468 2014-10-23 12:07:41,458Z - INFO - ESS it 1 -- LL train | last f = -669.63 -- test set error | last f = 0.77447 -- LL test | last f = -596.72 -- test set error (marginalized over f's)= 0.78298 -- average per-atom negative log posterior marginals = 2.0255 2014-10-23 12:07:41,602Z - INFO - ESS it 2 -- LL train | last f = -668.73 -- test set error | last f = 0.8 -- LL test | last f = -594.73 -- test set error (marginalized over f's)= 0.79149 -- average per-atom negative log posterior marginals = 2.0383 2014-10-23 12:07:41,739Z - INFO - ESS it 3 -- LL train | last f = -666.17 -- test set error | last f = 0.81277 -- LL test | last f = -597.84 -- test set error (marginalized over f's)= 0.79574 -- average per-atom negative log posterior marginals = 2.0085 2014-10-23 12:07:41,852Z - INFO - ESS it 4 -- LL train | last f = -653.86 -- test set error | last f = 0.78298 -- LL test | last f = -587.12 -- test set error (marginalized over f's)= 0.78723 -- average per-atom negative log posterior marginals = 2.017 2014-10-23 12:07:41,968Z - INFO - ESS it 5 -- LL train | last f = -645.26 -- test set error | last f = 0.75745 -- LL test | last f = -577.59 -- test set error (marginalized over f's)= 0.79149 -- average per-atom negative log posterior marginals = 1.9915 2014-10-23 12:07:42,046Z - INFO - ESS it 6 -- LL train | last f = -632.13 -- test set error | last f = 0.75319 -- LL test | last f = -566.05 -- test set error (marginalized over f's)= 0.78298 -- average per-atom negative log posterior marginals = 1.9574 2014-10-23 12:07:42,167Z - INFO - ESS it 7 -- LL train | last f = -633.45 -- test set error | last f = 0.68936 -- LL test | last f = -522.1 -- test set error (marginalized over f's)= 0.73617 -- average per-atom negative log posterior marginals = 1.8979 2014-10-23 12:07:42,236Z - INFO - ESS it 8 -- LL train | last f = -627.64 -- test set error | last f = 0.68511 -- LL test | last f = -512.06 -- test set error (marginalized over f's)= 0.7234 -- average per-atom negative log posterior marginals = 1.8383 2014-10-23 12:07:42,380Z - INFO - ESS it 9 -- LL train | last f = -620.22 -- test set error | last f = 0.65532 -- LL test | last f = -500.42 -- test set error (marginalized over f's)= 0.70213 -- average per-atom negative log posterior marginals = 1.7915
... and now we can run the same chain for another 10 samples if we wish:
n_data=10
# !rm -rf /tmp/pygpstruct_demo # don't erase the results folder this time !
prepare_from_data_chain.learn_predict_gpstruct_wrapper(data_indices_train = np.arange(n_data), # training data= first 10 data files
data_indices_test = np.arange(n_data, n_data*2), # test data= next 10 files
result_prefix='/tmp/pygpstruct_demo/', # where to store the result files
data_folder = pygpstruct_location + '/data/japanesene/',
n_samples=20, # how many MCMC iterations
task='japanesene',
prediction_thinning=1
)
!rm -rf /tmp/pygpstruct_demo # clean up afterwards however
2014-10-23 12:07:42,441Z - INFO - learn_predict_gpstruct started with arguments: {'kernel': <function kernel_linear_unary at 0x7f3c92ae3ed8>, 'console_log': True, 'random_seed': 0, 'prediction_verbosity': None, 'n_samples': 20, 'hp_sampling_thinning': 1, 'hp_sampling_mode': None, 'hp_debug': False, 'prior': 1, 'lhp_update': {}, 'prepare_from_data': <function <lambda> at 0x7f3c8c7c2758>, 'stop_check': None, 'n_f_star': 0, 'prediction_thinning': 1, 'result_prefix': '/tmp/pygpstruct_demo/'} 2014-10-23 12:07:42,443Z - INFO - prepare_from_data_chain started with arguments: {'data_folder': '/home/sb358/pygpstruct_demo/data/japanesene/', 'data_indices_train': array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]), 'n_labels': 17, 'native_implementation': False, 'data_indices_test': array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]), 'n_features_x': 102799, 'logger': <logging.Logger object at 0x7f3c8e3541d0>} 2014-10-23 12:07:42,537Z - DEBUG - prepare kernel matrices 2014-10-23 12:07:42,549Z - DEBUG - start MCMC chain 2014-10-23 12:07:42,612Z - INFO - hotstart from iteration 10, including stored random state 2014-10-23 12:07:42,717Z - INFO - ESS it 10 -- LL train | last f = -620.63 -- test set error | last f = 0.64255 -- LL test | last f = -498.84 -- test set error (marginalized over f's)= 0.70638 -- average per-atom negative log posterior marginals = 1.766 2014-10-23 12:07:42,854Z - INFO - ESS it 11 -- LL train | last f = -619.49 -- test set error | last f = 0.6383 -- LL test | last f = -494.85 -- test set error (marginalized over f's)= 0.68085 -- average per-atom negative log posterior marginals = 1.7191 2014-10-23 12:07:42,958Z - INFO - ESS it 12 -- LL train | last f = -617.51 -- test set error | last f = 0.64681 -- LL test | last f = -491.99 -- test set error (marginalized over f's)= 0.68511 -- average per-atom negative log posterior marginals = 1.7064 2014-10-23 12:07:43,091Z - INFO - ESS it 13 -- LL train | last f = -617.32 -- test set error | last f = 0.6383 -- LL test | last f = -492.88 -- test set error (marginalized over f's)= 0.6766 -- average per-atom negative log posterior marginals = 1.6766 2014-10-23 12:07:43,170Z - INFO - ESS it 14 -- LL train | last f = -617.32 -- test set error | last f = 0.6383 -- LL test | last f = -492.89 -- test set error (marginalized over f's)= 0.65957 -- average per-atom negative log posterior marginals = 1.6426 2014-10-23 12:07:43,263Z - INFO - ESS it 15 -- LL train | last f = -613.81 -- test set error | last f = 0.6383 -- LL test | last f = -487.01 -- test set error (marginalized over f's)= 0.65532 -- average per-atom negative log posterior marginals = 1.6383 2014-10-23 12:07:43,386Z - INFO - ESS it 16 -- LL train | last f = -613.32 -- test set error | last f = 0.64255 -- LL test | last f = -486.03 -- test set error (marginalized over f's)= 0.65106 -- average per-atom negative log posterior marginals = 1.6255 2014-10-23 12:07:43,531Z - INFO - ESS it 17 -- LL train | last f = -613.44 -- test set error | last f = 0.65957 -- LL test | last f = -488.65 -- test set error (marginalized over f's)= 0.65532 -- average per-atom negative log posterior marginals = 1.6 2014-10-23 12:07:43,676Z - INFO - ESS it 18 -- LL train | last f = -611.24 -- test set error | last f = 0.6383 -- LL test | last f = -483.63 -- test set error (marginalized over f's)= 0.65532 -- average per-atom negative log posterior marginals = 1.6043 2014-10-23 12:07:43,808Z - INFO - ESS it 19 -- LL train | last f = -610.76 -- test set error | last f = 0.64255 -- LL test | last f = -480.88 -- test set error (marginalized over f's)= 0.65532 -- average per-atom negative log posterior marginals = 1.5872
n_data=150
!rm -rf /tmp/pygpstruct_demo_long_run
prepare_from_data_chain.learn_predict_gpstruct_wrapper(data_indices_train = np.arange(n_data), # training data= first 10 data files
data_indices_test = np.arange(n_data, n_data*2), # test data= next 10 files
result_prefix='/tmp/pygpstruct_demo_long_run/', # where to store the result files
data_folder = pygpstruct_location + '/data/basenp/',
n_samples=1001, # how many MCMC iterations
task='basenp',
prediction_thinning=100,
native_implementation=True
)
2014-10-23 12:07:44,093Z - INFO - learn_predict_gpstruct started with arguments: {'kernel': <function kernel_linear_unary at 0x7f3c92ae3ed8>, 'console_log': True, 'random_seed': 0, 'prediction_verbosity': None, 'n_samples': 1001, 'hp_sampling_thinning': 1, 'hp_sampling_mode': None, 'hp_debug': False, 'prior': 1, 'lhp_update': {}, 'prepare_from_data': <function <lambda> at 0x7f3c8c794140>, 'stop_check': None, 'n_f_star': 0, 'prediction_thinning': 100, 'result_prefix': '/tmp/pygpstruct_demo_long_run/'} 2014-10-23 12:07:44,103Z - INFO - prepare_from_data_chain started with arguments: {'data_folder': '/home/sb358/pygpstruct_demo/data/basenp/', 'data_indices_train': array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149]), 'n_labels': 3, 'native_implementation': True, 'data_indices_test': array([150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299]), 'n_features_x': 6438, 'logger': <logging.Logger object at 0x7f3c8e3541d0>} 2014-10-23 12:07:45,524Z - DEBUG - prepare kernel matrices 2014-10-23 12:07:58,021Z - DEBUG - start MCMC chain 2014-10-23 12:07:58,345Z - INFO - ESS it 0 -- LL train | last f = -4015.8 -- test set error | last f = 0.70376 -- LL test | last f = -3924.2 -- test set error (marginalized over f's)= 0.70376 -- average per-atom negative log posterior marginals = 1 2014-10-23 12:08:04,689Z - INFO - ESS it 100 -- LL train | last f = -2789.6 -- test set error | last f = 0.32623 -- LL test | last f = -2861.9 -- test set error (marginalized over f's)= 0.32707 -- average per-atom negative log posterior marginals = 0.30886 2014-10-23 12:08:10,802Z - INFO - ESS it 200 -- LL train | last f = -2147.4 -- test set error | last f = 0.23935 -- LL test | last f = -2319.1 -- test set error (marginalized over f's)= 0.25084 -- average per-atom negative log posterior marginals = 0.34557 2014-10-23 12:08:17,073Z - INFO - ESS it 300 -- LL train | last f = -1761.2 -- test set error | last f = 0.19451 -- LL test | last f = -1965.9 -- test set error (marginalized over f's)= 0.20768 -- average per-atom negative log posterior marginals = 0.26654 2014-10-23 12:08:23,389Z - INFO - ESS it 400 -- LL train | last f = -1520.3 -- test set error | last f = 0.15387 -- LL test | last f = -1637.3 -- test set error (marginalized over f's)= 0.16788 -- average per-atom negative log posterior marginals = 0.22141 2014-10-23 12:08:29,564Z - INFO - ESS it 500 -- LL train | last f = -1331.6 -- test set error | last f = 0.13229 -- LL test | last f = -1407.2 -- test set error (marginalized over f's)= 0.13649 -- average per-atom negative log posterior marginals = 0.20095 2014-10-23 12:08:35,868Z - INFO - ESS it 600 -- LL train | last f = -1179.1 -- test set error | last f = 0.10706 -- LL test | last f = -1283.4 -- test set error (marginalized over f's)= 0.11996 -- average per-atom negative log posterior marginals = 0.17993 2014-10-23 12:08:42,163Z - INFO - ESS it 700 -- LL train | last f = -1041.9 -- test set error | last f = 0.092209 -- LL test | last f = -1113.2 -- test set error (marginalized over f's)= 0.10678 -- average per-atom negative log posterior marginals = 0.16059 2014-10-23 12:08:48,479Z - INFO - ESS it 800 -- LL train | last f = -954 -- test set error | last f = 0.085482 -- LL test | last f = -1062.7 -- test set error (marginalized over f's)= 0.091087 -- average per-atom negative log posterior marginals = 0.15863 2014-10-23 12:08:54,752Z - INFO - ESS it 900 -- LL train | last f = -891.07 -- test set error | last f = 0.088004 -- LL test | last f = -1003.9 -- test set error (marginalized over f's)= 0.087444 -- average per-atom negative log posterior marginals = 0.14882 2014-10-23 12:09:01,010Z - INFO - ESS it 1000 -- LL train | last f = -813.27 -- test set error | last f = 0.079877 -- LL test | last f = -980.94 -- test set error (marginalized over f's)= 0.081558 -- average per-atom negative log posterior marginals = 0.14546